Domain Adaptation-Based Deep Learning for Automated Tumor Cell (TC) Scoring and Survival Analysis on PD-L1 Stained Tissue Images.

We report the ability of two deep learning-based decision systems to stratify non-small cell lung cancer (NSCLC) patients treated with checkpoint inhibitor therapy into two distinct survival groups. Both systems analyze functional and morphological properties of epithelial regions in digital histopathology whole slide images stained with the SP263 PD-L1 antibody. The first system learns to replicate the pathologist assessment of the Tumor Cell (TC) score with a cut-point for positivity at 25% for patient stratification. The second system is free from assumptions related to TC scoring and directly learns patient stratification from the overall survival time and event information. Both systems are built on a novel unpaired domain adaptation deep learning solution for epithelial region segmentation. This approach significantly reduces the need for large pixel-precise manually annotated datasets while superseding serial sectioning or re-staining of slides to obtain ground truth by cytokeratin staining. The capacity of the first system to replicate the TC scoring by pathologists is evaluated on 703 unseen cases, with an addition of 97 cases from an independent cohort. Our results show Lin's concordance values of 0.93 and 0.96 against pathologist scoring, respectively. The ability of the first and second system to stratify anti-PD-L1 treated patients is evaluated on 151 clinical samples. Both systems show similar stratification powers (first system: HR = 0.539, p = 0.004 and second system: HR = 0.525, p = 0.003) compared to TC scoring by pathologists (HR = 0.574, p = 0.01).

Assessment of Immunological Features in Muscle-Invasive Bladder Cancer Prognosis Using Ensemble Learning.

The clinical staging and prognosis of muscle-invasive bladder cancer (MIBC) routinely includes the assessment of patient tissue samples by a pathologist. Recent studies corroborate the importance of image analysis in identifying and quantifying immunological markers from tissue samples that can provide further insight into patient prognosis. In this paper, we apply multiplex immunofluorescence to MIBC tissue sections to capture whole-slide images and quantify potential prognostic markers related to lymphocytes, macrophages, tumour buds, and PD-L1. We propose a machine-learning-based approach for the prediction of 5 year prognosis with different combinations of image, clinical, and spatial features. An ensemble model comprising several functionally different models successfully stratifies MIBC patients into two risk groups with high statistical significance (p value < 1×10-5). Critical to improving MIBC survival rates, our method correctly classifies 71.4% of the patients who succumb to MIBC, which is significantly more than the 28.6% of the current clinical gold standard, the TNM staging system.

Graph-based description of tertiary lymphoid organs at single-cell level.

Our aim is to complement observer-dependent approaches of immune cell evaluation in microscopy images with reproducible measures for spatial composition of lymphocytic infiltrates. Analyzing such patterns of inflammation is becoming increasingly important for therapeutic decisions, for example in transplantation medicine or cancer immunology. We developed a graph-based assessment of lymphocyte clustering in full whole slide images. Based on cell coordinates detected in the full image, a Delaunay triangulation and distance criteria are used to build neighborhood graphs. The composition of nodes and edges are used for classification, e.g. using a support vector machine. We describe the variability of these infiltrates on CD3/CD20 duplex staining in renal biopsies of long-term functioning allografts, in breast cancer cases, and in lung tissue of cystic fibrosis patients. The assessment includes automated cell detection, identification of regions of interest, and classification of lymphocytic clusters according to their degree of organization. We propose a neighborhood feature which considers the occurrence of edges with a certain type in the graph to distinguish between phenotypically different immune infiltrates. Our work addresses a medical need and provides a scalable framework that can be easily adjusted to the requirements of different research questions.

Automatic discovery of image-based signatures for ipilimumab response prediction in malignant melanoma.

In the context of precision medicine with immunotherapies there is an increasing need for companion diagnostic tests to identify potential therapy responders and avoid treatment coming along with severe adverse events for non-responders. Here, we present a retrospective case study to discover image-based signatures for developing a potential companion diagnostic test for ipilimumab (IPI) in malignant melanoma. Signature discovery is based on digital pathology and fully automatic quantitative image analysis using virtual multiplexing as well as machine learning and deep learning on whole-slide images. We systematically correlated the patient outcome data with potentially relevant local image features using a Tissue Phenomics approach with a sound cross validation procedure for reliable performance evaluation. Besides uni-variate models we also studied combinations of signatures in several multi-variate models. The most robust and best performing model was a decision tree model based on relative densities of CD8+ tumor infiltrating lymphocytes in the intra-tumoral infiltration region. Our results are well in agreement with observations described in previously published studies regarding the predictive value of the immune contexture, and thus, provide predictive potential for future development of a companion diagnostic test.

Automated tumour budding quantification by machine learning augments TNM staging in muscle-invasive bladder cancer prognosis.

Tumour budding has been described as an independent prognostic feature in several tumour types. We report for the first time the relationship between tumour budding and survival evaluated in patients with muscle invasive bladder cancer. A machine learning-based methodology was applied to accurately quantify tumour buds across immunofluorescence labelled whole slide images from 100 muscle invasive bladder cancer patients. Furthermore, tumour budding was found to be correlated to TNM (p = 0.00089) and pT (p = 0.0078) staging. A novel classification and regression tree model was constructed to stratify all stage II, III, and IV patients into three new staging criteria based on disease specific survival. For the stratification of non-metastatic patients into high or low risk of disease specific death, our decision tree model reported that tumour budding was the most significant feature (HR = 2.59, p = 0.0091), and no clinical feature was utilised to categorise these patients. Our findings demonstrate that tumour budding, quantified using automated image analysis provides prognostic value for muscle invasive bladder cancer patients and a better model fit than TNM staging.

Deep Semi Supervised Generative Learning for Automated Tumor Proportion Scoring on NSCLC Tissue Needle Biopsies.

The level of PD-L1 expression in immunohistochemistry (IHC) assays is a key biomarker for the identification of Non-Small-Cell-Lung-Cancer (NSCLC) patients that may respond to anti PD-1/PD-L1 treatments. The quantification of PD-L1 expression currently includes the visual estimation by a pathologist of the percentage (tumor proportional scoring or TPS) of tumor cells showing PD-L1 staining. Known challenges like differences in positivity estimation around clinically relevant cut-offs and sub-optimal quality of samples makes visual scoring tedious and subjective, yielding a scoring variability between pathologists. In this work, we propose a novel deep learning solution that enables the first automated and objective scoring of PD-L1 expression in late stage NSCLC needle biopsies. To account for the low amount of tissue available in biopsy images and to restrict the amount of manual annotations necessary for training, we explore the use of semi-supervised approaches against standard fully supervised methods. We consolidate the manual annotations used for training as well the visual TPS scores used for quantitative evaluation with multiple pathologists. Concordance measures computed on a set of slides unseen during training provide evidence that our automatic scoring method matches visual scoring on the considered dataset while ensuring repeatability and objectivity.

Tissue Phenomics for prognostic biomarker discovery in low- and intermediate-risk prostate cancer.

Tissue Phenomics is the discipline of mining tissue images to identify patterns that are related to clinical outcome providing potential prognostic and predictive value. This involves the discovery process from assay development, image analysis, and data mining to the final interpretation and validation of the findings. Importantly, this process is not linear but allows backward steps and optimization loops over multiple sub-processes. We provide a detailed description of the Tissue Phenomics methodology while exemplifying each step on the application of prostate cancer recurrence prediction. In particular, we automatically identified tissue-based biomarkers having significant prognostic value for low- and intermediate-risk prostate cancer patients (Gleason scores 6-7b) after radical prostatectomy. We found that promising phenes were related to CD8(+) and CD68(+) cells in the microenvironment of cancerous glands in combination with the local micro-vascularization. Recurrence prediction based on the selected phenes yielded accuracies up to 83% thereby clearly outperforming prediction based on the Gleason score. Moreover, we compared different machine learning algorithms to combine the most relevant phenes resulting in increased accuracies of 88% for tumor progression prediction. These findings will be of potential use for future prognostic tests for prostate cancer patients and provide a proof-of-principle of the Tissue Phenomics approach.

Image-based characterization of thrombus formation in time-lapse DIC microscopy.

The characterization of thrombus formation in time-lapse DIC microscopy is of increased interest for identifying genes which account for atherothrombosis and coronary artery diseases (CADs). In particular, we are interested in large-scale studies on zebrafish, which result in large amount of data, and require automatic processing. In this work, we present an image-based solution for the automatized extraction of parameters quantifying the temporal development of thrombotic plugs. Our system is based on the joint segmentation of thrombotic and aortic regions over time. This task is made difficult by the low contrast and the high dynamic conditions observed in vivo DIC microscopic scenes. Our key idea is to perform this segmentation by distinguishing the different motion patterns in image time series rather than by solving standard image segmentation tasks in each image frame. Thus, we are able to compensate for the poor imaging conditions. We model motion patterns by energies based on the idea of dynamic textures, and regularize the model by two prior energies on the shape of the aortic region and on the topological relationship between the thrombus and the aorta. We demonstrate the performance of our segmentation algorithm by qualitative and quantitative experiments on synthetic examples as well as on real in vivo microscopic sequences.

Joint thrombus and vessel segmentation using dynamic texture likelihoods and shape prior.

The segmentation of thrombus and vessel in microscopic image sequences is of high interest for identifying genes linked to cardiovascular diseases. This task is however challenging because of the low contrast and the highly dynamic conditions observed in time-lapse DIC in-vivo microscopic scenes. In this work, we introduce a probabilistic framework for the joint segmentation of thrombus and vessel regions. Modeling the scene with dynamic textures, we derive two likelihood functions to account for both spatial and temporal discrepancies of the motion patterns. A tubular shape prior is moreover introduced to constrain the aortic region. Extensive experiments on microscopic sequences quantitatively show the good performance of our approach.

Characterization of the hemodynamic response in the rat lumbar spinal cord using intrinsic optical imaging and laser speckle.

Quantifying spinal cord functions is crucial for understanding neurophysiological mechanisms governing the intact and the injured spinal cord. Intrinsic optical imaging (IOI) and laser speckle provides measures of deoxyhemoglobin (HbR) and oxyhemoglobin (HbO(2)) concentrations, blood volume (BV) and blood flow (BF) at high spatial and temporal resolution. In this study we used IOI and laser speckle to characterize the hemodynamic response to neuronal activation in the lumbar spinal cord of anaesthetized rats (N=9). We report consistent temporal variations of HbR, HbO(2), BV and BF located ipsilaterally at L3-L5. Responses were significantly higher when stimulation intensity was increased. Vascular changes extended several millimetres from the epicenter, supporting the venous drainage observed in functional magnetic resonance imaging studies.

Optical imaging of vascular and metabolic responses in the lumbar spinal cord after T10 transection in rats.

Neuronal and vascular reorganization after spinal cord injury (SCI) is scarcely known although its characterization has major implications in understanding the functioning of the altered spinal cord. Several electrophysiological and anatomical lines of evidence support plasticity caudal to the lesion site, but do not provide sufficient clues about neuronal and vascular reorganization after SCI. The aim of the present study was to compare neuronal activation in the lumbar spinal cord between uninjured and SCI rats with novel optical imaging technology. The results showed significant haemodynamic response differences after sciatic nerve stimulation in uninjured controls, in comparison to SCI rats. Both timing and shape of the response were modified. In uninjured rats, blood flow presented an initial dip but was rapidly drained from the activation site through the venous system. In comparison, the blood transfer rate in SCI rats was much slower. Damaged blood vessels at the lesion site after thoracic SCI impacted the vascular response upon neuronal activation in the lumbar spinal cord. This observation is important in the study of spinal cord function after SCI by imaging techniques based on haemodynamics (blood oxygenation level-dependent using functional magnetic resonance imaging (BOLD fMRI) and optical imaging). In conclusion, our results indicate that new avenues quantifying the influence of vascular plumbing will have to be developed to explore the efficacy of rehabilitation and pharmacological therapies by haemodynamic imaging.